User-Space Solutions to Thread Switching Overhead

Writing communication software, which spends a significant amount of time on handling incoming messages, is difficult: Active Messages are fast, but awkward to use; traditional popup threads are easy to use, but expensive. Therefore, we have implemented a portable, hybrid upcall mechanism that is easier to use than Active Messages, yet avoids unnecessary thread switching. We allow upcalls to block on locks protecting shared data, but disallow blocking on synchronous communication and condition variables. To deal with the latter restriction we use continuations as in [Draves91]. We have incorporated this hybrid upcall mechanism into the runtime system of Orca,an object-based Distributed Shared Memory system. This paper describes the modifications and the performance improvement obtained with the restricted upcall model and continuations. Comparison with the previous version of the Orca runtime system, which uses blocking upcalls, shows that the code restructuring has improved communication performance and reduced memory consumption.